feat(validation): P3-3 OOS stability validation (equal_weight vs ic_weighted) - #12
Merged
Merged
Conversation
…eighted) A REPORT-ONLY validation layer -- no new alpha complexity, no change to portfolio / execution / factor math. Validates whether the P3-1 equal-weight and P3-2 ic-weighted combinations are stable out of sample before adding any further alpha sophistication. Runner (qt/oos_stability.py, CLI run-phase3-oos): - ONE shared data load + ONE processed factor panel; the backtest runs twice (EqualWeightAlpha / RollingICWeightAlpha) under identical universe, neutralization, portfolio and execution rules. - Every diagnostic is split at oos.split_date: train = [start, split), test = [split, end]; subperiod navs are rebased so nothing bleeds across. - Split semantics are walk-forward (rolling subperiod): weights at any date use only observations realized by that date (t + h <= d). A split-boundary test proves perturbing EVERY post-split forward return leaves all train-period weights bit-identical. Freeze-at-split is deliberately NOT added (it would be a new alpha mode, out of scope). - Guards: requires data.source=tushare and an 'oos' config section; the split date must lie strictly inside the data window (config-validated). Report (phase3_oos_stability.md): split boundaries (exact realized dates + day counts), per-subperiod performance for both models (annual / vol / sharpe / maxDD / turnover / rebalances), per-series IC stability (mean / IR / hit rate / n + train-vs-test sign consistency) for every raw factor and both combo scores, ic-weighted weight stability (per-rebalance weights labelled train/test, sign flips on trained rows, fallback count + reasons), and the explicit small-sample / not-a-return-claim caveat. Real run (SSE50, 2y 2022-07~2024-06, split 2023-07-01, 77 names, ~16 min) -- the honest headline is INSTABILITY: - all three raw factors FLIP IC sign train->test (momentum_20 -0.024->+0.006, roe -0.029->+0.007, netprofit_yoy -0.010->+0.005; sign consistency NO); hit rates 46-53% (coin-flip range); - ic_weighted weights flip sign 7/3/4 times across 23 rebalances; - performance: equal_weight train -6.81% / test -5.27%; ic_weighted train -1.69% / test -2.70%. ic_weighted edges equal_weight in BOTH subperiods, but with near-zero, sign-flipping ICs this must NOT be read as skill -- the P3-2 one-year result does not extrapolate, which is exactly what this validation layer was built to surface. Regressions: phase3 equal_weight rerun -9.05% / IC 0.0083 and ic_weighted rerun -3.57% both unchanged; demo unchanged (ic 0.96 / annual 0.84); the report contains no token / secret-file content (scanned). Tests: 282 passed (was 269): +13 OOS (split no-leakage, subperiod slicing, IC period stats, sign consistency, weight sign flips excl. fallback rows, fallback-reason aggregation, config + guard + render contracts).
Review HIGH fixes on PR #12: 1. Subperiod performance was sliced by the nav row's SIGNAL date, but a row's return covers the holding window [rebalance, next rebalance] -- with split=2023-07-01 the 2023-06-30 row (held into July) credited test-period return to train. Now split_nav_by_holding assigns train <=> holding END on/before the split, test <=> holding START on/after it; straddling rows (incl. the last row, whose end is the unknown terminal candidate) are EXCLUDED from both and disclosed in the report. IC stats are likewise sliced by the realization date (t + horizon), not the factor date alone. Real impact (rerun, 929s): train eq -6.81% -> -11.92%, ic -1.69% -> -8.31% (the old train was polluted by post-split July returns); test unchanged (-5.27% / -2.70%); IC sign-flip finding unchanged (all NO). 2. run-phase3-oos now REQUIRES alpha.model='ic_weighted': any other model would silently run equal_weight twice and label one leg ic_weighted (a fake comparison). Readable ValueError + network-free test. Tests: 285 passed (was 282): +2 holding-window slicing (straddler + unknown -end), +1 fake-comparison guard; ic_period_stats test rewritten for realization-date slicing; render test asserts the straddler disclosure. Docs (RUNBOOK/TEST_REPORT/CLAUDE/AGENTS) updated with the new semantics and corrected real numbers.
…able Review LOW on PR #12: the per-rebalance weights table labelled the straddling 2023-06-30 row 'train' while the performance slicing had already excluded it from both subperiods as a boundary row. The period label in the weights table now matches the performance slicing (boundary, not train/test). Display-only; no metric changes. Render test asserts the boundary label.
…er review Doc-only sync in CLAUDE.md + AGENTS.md (both files, same drift): - merged-PR lists: add #10 (progress docs) and #11 (P3-2). - P3-2 entry: 'branch, awaiting acceptance' -> 'PR #11 merged to main'. - P3-3 entry: 'branch, awaiting acceptance' -> 'PR #12 OPEN, review HIGHs+LOW fixed, awaiting acceptance/merge'.
Maintaining two slightly-divergent progress docs invited drift (three review findings so far were exactly this). AGENTS.md is now a straight copy of CLAUDE.md; future updates edit CLAUDE.md and copy over.
StackOverFlow11
added a commit
that referenced
this pull request
Jun 10, 2026
docs: mark PR #12 (P3-3) merged in progress docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
P3-3 — OOS stability validation. Before adding any further alpha sophistication, this PR validates whether the P3-1
equal_weightand P3-2ic_weightedcombinations are stable out of sample. It is a report-only validation layer: no new alpha complexity, no change to portfolio / execution / factor math.Implementation
qt/oos_stability.py+ CLIrun-phase3-oos: ONE shared data load and ONE processed factor panel; the backtest runs twice (EqualWeightAlpha / RollingICWeightAlpha) under identical universe / neutralization / portfolio / execution rules.oos.split_date(new optional config section, cross-validated to lie strictly inside the data window) — train[start, split), test[split, end]; subperiod navs are rebased so no drawdown/annualization bleeds across the boundary.config/phase3_real_oos_stability.yaml: SSE50 extended to TWO years (2022-07 ~ 2024-06), split 2023-07-01 → train 1y / test 1y. The test year equals the phase3 baselines' window so numbers line up.Lookahead boundary
Evaluation is walk-forward (rolling subperiod): weights at any date d use only observations realized by d (
t + h <= d, the P3-2 property). The new split-boundary no-leakage test perturbs EVERY post-split forward return and asserts all train-period weights are bit-identical — no test-period return can reach a train-period computation. Freeze-at-split was deliberately NOT added (a new alpha mode = new alpha complexity, explicitly out of scope).Real run (77 names, 2 years, ~16 min) — the honest headline is INSTABILITY
Subperiod performance (same data, same rules):
equal_weightic_weighted(11 + 11 rebalances; the 2023-06-30 rebalance — whose holding window straddles the split — is excluded from both subperiods and disclosed in the report.)
IC stability — the key finding:
momentum_20roenetprofit_yoycombo_ic_weightedWeight stability: sign flips across 23 rebalances — momentum_20 ×7, roe ×3, netprofit_yoy ×4; 20 equal-weight fallbacks (all "insufficient realized IC history", before the 60d window filled).
Conclusion (disclosed in the report): all three raw factors FLIP IC sign train→test with coin-flip hit rates;
ic_weightededgesequal_weightin both subperiods but with near-zero, sign-flipping ICs this must NOT be read as skill — the P3-2 one-year result does not extrapolate. Surfacing exactly this was the purpose of the validation layer. NOT a return claim.Regressions & secret scan
run-phase0.config.jsoncontentQuality gates
pytest -p no:cacheproviderruff check .validate-config×6 (all old + phase3_real_oos_stability)Tests (network-free)
test_oos_stability.py(13): split-boundary no-leakage (perturb all post-split returns → train weights bit-identical), strict subperiod slicing + rebased nav + empty-slice NaN, IC period stats (mean/IR/hit-rate/n), nonzero-same-sign consistency, weight sign flips on trained rows only (fallback rows excluded), fallback-reason aggregation, OOS config validation + split-inside-window ConfigError, demo-source + missing-oosguards, report renders boundaries/metrics/stability/caveat with no secret.Docs
RUNBOOK (P3-3 section), TEST_REPORT (282 + breakdown), CLAUDE.md / AGENTS.md progress (incl. the instability finding).
Review fixes (0035a04)
Two HIGH findings, both fixed + real rerun:
split_nav_by_holdingassigns train ⇔ holding END ≤ split, test ⇔ holding START ≥ split; straddlers (and the unknown-end last row) are excluded from both and disclosed; IC stats slice by the realization date (t+horizon). Real impact: train eq −6.81%→−11.92%, ic −1.69%→−8.31% (the correction magnitude is itself evidence of the boundary bug); test unchanged; the IC sign-flip finding unchanged (all NO).run-phase3-oosnow refusesalpha.model != 'ic_weighted'(it would silently run equal_weight twice and label one leg ic_weighted). Readable ValueError + network-free test.Gates after fixes: 285 passed / ruff clean / OOS real rerun 929s with the corrected report.
boundary(wastrain), matching the performance slicing. Display-only; locked by a render test.